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ABSTRACT 

This document unites two comprehensive policy statements by 
the National Educational Research Policy and Priorities Board. "Investing in 
Learning" was first published in June 1999 and the followup, "Investing in 
Research," is published for the first time in this volume. "Investing in 
Learning" follows a thematic approach in establishing research goals and 
Qj^jgQtives that will support a high level of achievement for all students. 
Recommendations are grouped into these areas: (1) priorities are set and 

activities are problem centered; (2) high standards of quality are created 
and upheld; (3) work is collaborative and rigorous; (4) design for rigorous 
research; and (5) mission is congruent with resources. "Investing in 
Research" takes a more pragmatic approach in pointing out where improvements 
might be made in existing institutions and where new arrangements are needed 
to promote knowledge development and to apply research findings to high 
p 2 fiority areas. This set of policy recommendations addresses the urgency of 
increased rigor in education research, questions about the federal 
government's organizational arrangements, and the policy setting and 
leadership functions for research. (SLD) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 
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Dear Colleague: 

The present document unites in a single volume two comprehensive policy 
statements by the National Educational Research Policy and Priorities Board. 
Irwestingin Leamir^'wdtS first published in June 1999, and its follow-up, Imestingin 
Researdo, is being released today. Together these papers summarize the findings 
and conclusions of the Board with regard to the federal role in support of 
educational research. 

Since 1995, the 15-member Board has worked collaboratively with the Assistant 
Secretary for the Office of Educational Research and Improvement (OERI) to 
forge a national consensus with respect to a long-term agenda for educational 
research, development, and dissemination. We have come to a point at which the 
agency’s authorizing legislation is under review by the Congress, and when a new 
administration is imminent. Therefore, we feel a distinct obligation, based on our 
experience, to present our recommendations as a blueprint for progress. 

Investing in Learning follows a thematic approach in establishing research goals and 
objectives that will support a high level of achievement for all students. Irwestmg 
in Reseanh takes a more programmatic approach in pointing out, in the first place, 
where improvements might be made in existing institutions, and secondly, where 
new arrangements are needed both to promote knowledge development and to 
apply research findings to high priority areas. 

Readers are encouraged to respond and to address their comments to 

The National Educational Research Policy and Priorities Board 
80 F Street NW, Suite 100 
Washington, DC 20208-7564 
E-mail: nerppb(5)ed.gov 



Sincerely, 
Kenji Hakuta 



United States Department of Education □ Office of Educational Research and Improvement 
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Foreword 



The National Educational Research Policy and Priorities Board (NERPPB) is 
charged by Congress to advise the United States on the federal educational 
research and development effort. Since its establishment by the Educational 
Research, Development, and Improvement Act of 1994, the Board has under- 
taken systematic investigation on the dimensions and scope of educational 
research and development, much of which has been carried out under the 
auspices of the Office of Educational Research and Improvement (OERI) of the 
U.S. Department of Education and by its predecessor, the National Institute of 
Education. We have emerged from this study more firmly convinced than ever 
of the importance of sound, cumulative research in education. 

At the same time, we must acknowledge that there are serious shortcomings in 
the present research and development system in education-in its funding, 
structure, organization, approaches, and even its goals and objectives. Apart 
from our own firsthand observations of OERI and other federal agencies, we 
have consulted widely with the key stakeholders-educators, families, and 
researchers— about the changes and improvements that will correct our course 
and lead us to the advancements that we seek. 

This document is the first comprehensive statement on research in education 
that draws on our own systematic inquiry over the past 4 years. We have made 
recommendations for legislative and administrative changes that we are con- 
vinced are necessary to improve the research, development, and communication 
of research activities sponsored by the federal government. We hope these 
findings will be helpful and constructive, particularly in light of the consider- 
ation by this Congress of the authorization of the Office of Educational Re- 
search and Improvement. 

Readers are encouraged to respond and to address their comments to National 
Educational Research Policy and Priorities Board, 80 F Street NW, Suite 100, 
Washington, DC 20208-7564; E-mail: nerppb@ed.gov 

Kenji Hakuta 
Chair 

National Educational Research Policy and Priorities Board 
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Investing in Learning: 
A Policy Statement 
with Recommendations 
on Research in Education 



Summary of Recommendations 

Goal: Priorities Are Set and Activities Are Problem- 
Centered 

1. Student achievement— 'X\\e. priority for research in education must be a 
high level of achievement for all students, and, within that domain, the 
initial emphasis should be on reading and mathematics achievement. 

2. Reading, second language learning, and mathematics— Kecenx. reports from 
the National Research Council, Preventing Reading Difficulties in Young 
Children, and Improving Schooling for Language-Minority Children: A 
Research Agenda, synthesize strong bodies of research knowledge. A 
similar study on mathematics is currently under way. In each case, 
research is needed now to analyze how the results of our knowledge can 
be implemented in school programs and what factors lead to success or 
to difficulties. In reading, research is needed on how students become 
facile at reading complex text as they transition to advanced academic 
subjects such as history, social science, mathematics, and science. Re- 
search in both short- and long-term effects of specific education interven- 
tions for English language learners is needed, as well as techniques of 
assessment to measure competence, and transition points, from the first 
oral language to English; from oral language to literacy; and from lit- 
eracy to the academic discourse of specific disciplines. In mathematics, 
research is needed on why students have so much trouble making transi- 
tions (e.g., from concrete objects to more abstract ideas), understanding 
formal representations, multiplicative reasoning, and essential math- 
ematical and statistical concepts, such as chance, randomness, and prob- 
ability. 

3. Organization for learning out of school— To take advantage of different 
learning environments in which children from impoverished back- 
grounds often display more competence than in school settings, research 
is needed to design and test different models of after-school and summer 
programs to motivate, engage, and benefit children of low-income 
families. Work is also needed on types and features of after-school 
opportunities that most effectively motivate academic achievement and 
positive self-estimations; and how to design and test different models of 



Investing in Learning 



ERIC^ 



14 



Summary of Recommendations 



collaboration between schools and community groups dedicated to 
providing strong learning environments for poor children. 

4. Organization for learning in school — Retention, pull-out remediation, 
tracking, and segregated special education programs that stratify by race, 
class, and gender opportunities to learn do not result in high achieve- 
ment for all students. A more complete inventory of knowledge about 
effective practices for teaching academically challenging curricula with 
groups is needed, both for school populations in general, and for hetero- 
geneous groups in particular. Research is needed on questions of time for 
children to master challenging curricula, supportive school structures, 
and expectations for the breadth and depth of content. Within each of 
these is the question, do students from middle-income families, as well as 
from low-income, ethnic, and linguistic minority backgrounds, benefit 
from each organizational practice? An important area of inquiry is 
whether there are academic benefits to classroom diversity— does diver- 
sity improve subject-matter learning? 

5. Linking changes in teaching practice with improved student learning-^ 
Information is needed that can guide teachers and institutions who want 
to change their educational practice, particularly to reduce inequities in 
the opportunities of students who differ in socioeconomic status, ethnic 
background, and gender to learn successfully. This is especially impor- 
tant regarding the achievement of deep intellective competence advo- 
cated in current educational reforms. Such research would examine 
fundamental issues about the nature of teaching and learning, including, 
but not limited to, the importance of the skills and knowledge of teach- 
ers. We need to expand our knowledge and understanding of teaching 
practices, including teaching tools, such as assessment, that are successful 
with students who bring different cultural resources to their own and 
others’ learning. Research would examine, much more so than in the 
past, issues of what it takes to do effective and successful teaching with 
diverse populations of students. 

6. Linking teachers' professional development and teaching practices— 
search is needed to understand what effective teachers do and how they 
do it. Successful teaching involves not only the exercise of skills and 
application of knowledge but also flexible impfovisational adaptation in 
the classroom. Research is needed to understand the roles of increased 
knowledge and comprehension of subject-matter concepts and methods, 
the role of thorough understanding of the curriculum in the subject both 
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at the level one is teaching and in relation to other disciplines and 
grade levels, as well as the role of understanding students’ learning in 
improving teaching practice. Research should also investigate how 
the structure of teachers’ work supports or hinders their “on-the- 
job” learning and what kinds of abilities are learned in particular 
situations that can transfer to other settings. 

7. Understanding and supporting successful professional development— 
There is need for a better understanding of teachers’ development as 
“professional learning,” and of teaching as a “learning profession.” 
The model of learning how to teach, which is prevalent today, 
namely, that knowledge goes in during teacher education or profes- 
sional development and then comes out in the teachers’ own class- 
rooms, does not account for the engagement of teachers themselves 
in improving the practice of their profession. What teachers need to 
learn to put reforms in place is not separable from their actual 
teaching practices or from the development trajectories of their 
careers. Research needs to examine ways in which people of diverse 
cultural, ethnic, and socioeconomic backgrounds are attracted to 
careers in teaching. How can professional development resources 
help support diversity in the teaching profession as well as improve 
practice? Further research on teachers’ communities of practice is 
needed, building on Hndings that norms of responsibility and colle- 
gial efforts at professional problem solving are the most critical 
factors in improvement of teaching and learning. 

Goal: High Standards of Quality Are Created 
and Upheld 

8. Standing panels—St&ndxa^ panels should be established to review 
proposals for each OERI institute. These would be comprised of 25 
to 30 members, but with some overlapping membership, so that 
problems that cross boundaries can receive informed attention and 
that members of one panel with special knowledge could be invited 
to serve on another. 

9. Panel membership— Pineh should represent a broad range of perspec- 
tives. They must include members with strong disciplinary and 
methodological expertise. Across OERI panels, gender, race, 
ethnicity, and geographic diversity must be respected. Panel mem- 
bers should generally be nationally recognized figures. 
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10. Standards for panelists— The Board continues to support peer review 
standards adopted by OERI, with Board approval, which stipulate that 
all reviewers meet three criteria: demonstrated expertise, including 
training and experience in the subject area of the competition; in-depth 
knowledge of policy or practice in education; and in-depth knowledge of 
theoretical perspectives or methodological approaches in the subject area 
of the competition. 

1 1. Distinguishing between field-initiated and directed competitions— OT31 
should distinguish between field-initiated competitions and those that 
are directed, rather than trying to combine the two. 

12. Funding for peer review— The allowable percent of funds to support peer 
review should be increased so that the necessary standing panels and 
logistical support can be provided. 

13. Definitions— The term “research” should be more narrowly defined than 
it is in the 1994 law so that the boundaries of focused competitions for 
research can be limited. Research ishould emphasize basic research in 
education as well as investigations, experiments, and inquiry to develop 
new knowledge or apply tested knowledge. It should exclude develop- 
ment, planning, and demonstrations. The term “national significance” 
needs to be clarified through regulations or in legislative language so that 
reviewers understand that it includes research opportunities, not only 
important problems identified by educators. 

Goal: Work Is Collaborative and Rigorous 

14. Collaboration across federal agencies— The Assistant Secretary for OERI 
should extend efforts to join with other federal agencies, and perhaps 
foundations, to collaborate on common agendas. 

15. Coordination of research within the U.S. Department of Education— The 
Secretary should encourage, and the Assistant Secretary for OERI 
should provide special attention to performing a visible and constructive 
role in collaboration and coordination of education research within the 
Department. 

16. Linking ongoing research and practice-related efforts— 'Xhe Assistant Secre- 
tary should develop constructive means through which OERI, the 
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Research and Development Centers, and the Regional Educational 
Laboratories can function closely together to maximize their collec- 
tive impact. 

17. Synthesis activities— OE^ should support synthesis activities across 
all-important fields of educational research, summing up progress 
continually and drawing implications for policy and practice. 

18. OERI staff— Assistant Secretary needs to determine the responsi- 
bilities most appropriate to accomplish the research functions of 
OERI, both to advance its work and to attract, retain, and continu- 
ously nurture staff with the requisite training and opportunities. 

19. Collaborative problem-solving research and development— OEEl 
should begin development of a new design for research that would 
focus explicitly on solving specific current problems of practice and 
at the same time be accountable for developing and testing general 
principles of education that can “travel” to locations beyond those in 
which the research is done The central idea is to develop a system of 
support for projects in which professional researchers and profes- 
sional educators share in the accountability for achieving success in 
improving educational practices and outcomes. These projects may 
also include program developers, curriculum specialists, or policy 
specialists. Initially, this effort might be launched through a “work- 
ing group” to assist in designing specific parts of the priority research 
agenda for which this model would be best suited, the role of OERI 
and outside contractors or grantees, coordination across multiple 
sites, expectations as to scale and length of commitment and the like. 

Goal: Mission Is Congruent with Resources 

20. Funding research in education— VxaKXmg for educational research 
must be increased dramatically. An interim target should be to reach 
the level of 0.5 percent of our nation’s expenditure for elementary 
and secondary education, about $1.5 billion annually, which was the 
amount proposed by the President’s Committee of Advisors on 
Science and Technology. It would be feasible to reach this target 
over a 5-year period. 
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21. Aligning resources and mission— Over the next 5 years, OERI should 
work to make the resources and missions better aligned. The missions 
must be matched with money. 

22. Research supervision—Some focal point should be created by Congress 
for research leadership that can span across administrations. 

23. Allocation requirements— T\it allocation provisions for institutes and for 
types of support in the Educational Research, Development, Dissemina- 
tion, and Improvement Act of 1994 should be removed. 
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I. Introduction 



Research in Education 

America’s students fall far short of academic achievement levels that policy 
leaders, parents, the public, and the media believe are necessary to equip them 
for living, for active citizenship, and for productive employment in the 21st 
century. This is perhaps the most frequently repeated message Americans hear, 
see, or read about education. Rarely, however, is our nation told that research 
has proved that it can make a difference in the practice of education so that 
more students will learn effectively. Yet that has been the record. Members of 
the National Educational Research Policy and Priorities Board believe that 
investments in education research are justified, and, more important, that 
commitment to substantial growth in federal support of these investments is 
urgent. Without these investments, guidance for education policies and practices 
is too often left to uninformed opinion and unreasoned prejudice. 

The Board has reached these conclusions as a result of experiences shared by the 
members since its establishment in March 1995. It has carried out the duties 
assigned in law, many of them in collaboration with OERI, commissioned 
studies and evaluations, and consulted with many individuals. Representatives 
of scholarly organizations, schools, advocacy groups, and governmental agen- 
cies have briefed the Board about how research is carried on, what findings have 
been reported and analyzed, and how those findings have been put to use. 
Established under the Educational Research, Development, Dissemination, and 
Improvement Act of 1994, the Board has had responsibility to develop policies 
and priorities for research in education. This policy paper is the Board’s first 
comprehensive statement on the federal effort in education research. 

Two important trends provide the context for the nation’s challenge to helping 
our youth reach acceptable educational performance standards and outcomes. 
The first of these is the rapid increase in the number of students “at risk” in 
school districts that are least able to marshal the human and financial resources 
to meet their needs. The demographic and geographic characteristics of pro- 
jected growth in the youth population over the next 30 years suggest that 
virtually all of it will be concentrated in these “at risk” areas. But one should 
not suppose that our national challenge is confined to children at risk in dete- 
riorating urban school systems. As the recent Third International Mathematics 
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and Science Study (TIMSS) has made clear, deficiencies in mathematics and 
science skills and knowledge extend to all students at all levels of relative profi- 
ciency, including those who live in well-financed districts with mostly majority 
population. 

A second important contextual trend is the accelerating onrush of information 
technology. Already reaching into many aspects of the lives of students, the 
new technologies are increasingly shaping formal education, for better or 
worse, and reemphasizing disparities between the “haves” and the “have-nots.” 
This trend is not simply a matter of access to computer hardware and software, 
vital as these factors are; it is also about the critical need to plan and integrate 
new technology into teaching and curricula, so as to expand and extend student 
learning. 

These interacting trends represent a problem of immense national significance. 
The national educational enterprise in its many forms is widely and correctly 
understood to be a central device for the development of the knowledge, skills, 
and perspectives necessary to the success of our economy and the well being of 
our society. Education is also part of the glue that helps to bind us together as a 
community and a bridge across our many differences. Meeting these new chal- 
lenges requires more than good will, energy, and resources. It also requires 
putting what we know to work and expanding our knowledge base so that our 
capacity to meet the challenges will be expanded. Trying to implement our 
hopes and our goals without careful research, testing, and development's likely 
to increase our frustrations without improving our performance. 

In no way is research in education a quick and effortless path to success. Over 
the years, there have been many hard lessons to be learned. Educational im- 
provement occurs slowly and in small increments no matter how powerful the 
research base behind it. Deep disagreements among prominent researchers are 
continual and perhaps inevitable. Professional educators have rarely become 
enthusiastic consumers of research. Weak designs and measures, combined with 
these professional doubts and disputes, have produced too many research results 
whose values and political implications are more prominent than their scientific 
validity. The educational research system has had powerful constraints and 
limitations on it, which have hindered numberless researchers and projects. The 
wonder is that educational researchers have been able to accomplish what they 
have. 
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Yet the accomplishments of research in education have been significant and 
their influence on education often wide. This perspective has been summa- 
rized in the report from a June 1998 conference on National Directions in 
Education Research Planning, which the Board sponsored jointly with 
OERI: 

Educational research has been used time and again, at critical junc- 
tures, to improve teaching and learning. Important examples range 
from John Dewey through constructivism, to Edward L. Thorndike 
through behaviorism and educational testing. . . 

Education research has supported the design and evaluation of many gov- 
ernmental programs at all levels. Studies of learning and school organiza- 
tions have had a major impact on teaching, assessment, and education 
reforms. Three recent reports from the National Research Council (NRC) 
that sum up what has already been learned and how it might be used in 
education include: Preventing Reading Difficulties in Young Children-, Im- 
proving Schooling for Language-Minority Children: A Research Agenda; and 
How People Learn: Brain, Mind, Experience, and School. A recent, multifac- 
eted set of research reports, the Third International Mathematics and Sci- 
ence Study (TEMSS), is the latest, most comprehensive, and most significant 
of a generation’s progress in building comparative international assessments 
of learning and instruction. The widely acclaimed “Success for All” and 
projects of the New American Schools are demonstrating the practical and 
powerful effects that research can have when it is systematically applied in 
the classroom. 

Our first conclusion from these and many other examples is that research has 
a proven record in education. Our second is that federal support for education 
research is an investment that must be expanded several-fold. Others have 
come to this second conclusion as well. For example, in a 1997 report, a 
panel of the President’s Committee of Advisors on Science and Technology 
(PCASTJ called for sharp increases in education research appropriations— 
reaching 0.5 percent of U.S. spending for elementary and secondary educa- 
tion. The report of the National Directions conference agreed that greater 
funding for research in education would be wise and productive: 

The prospect is that more confidence and opportunity could pay off 
handsomely, if the support is strategically provided. 
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The National Educational Research Policy and Priorities Board not only con- 
curs, it has created this policy statement to describe critical elements of a strate- 
gic design for the federal government’s role in education research. 

Role of the National Educational Research Policy 
and Priorities Board 

Congress created the Board as an external policy setting and advisory body for 
research in education. The Board sets priorities and approves standards. It also 
conducts reviews of OERI work, serves in a liaison capacity with the education 
research field and the public, and has responsibilities to strengthen the educa- 
tion research and development system^ It enjoys substantial independence in 
gathering information, commissioning consultants, meeting with representa- 
tives of the education research system and consumers of that system, and, 
perhaps most importantly, communicating with the American public and 
Congress about education research. Among its specific responsibilities are the 
following: 

• provide guidance to Congress in its oversight of OERI; 

• review regularly, evaluate, and publicly comment upon, the imple- 
mentation of Board recommended priorities and policies by the 
Department md the Congress; and 

• advise the (people of the) United States on the federal educational 
research and development effort. 

The operations of the Board are required to be collaborative— carried out in 
concert with the Assistant Secretary as well as vrith researchers, teachers, school 
administrators, parents, students, employers, and policymakers. Indeed, the - 
concept of collaboration is emphasized both in the statement of Board responsi- 
bilities in the law; (such as in policy and priorities setting) and in the categories 
for Board appointments mandated in the law, namely: 

• education researchers, nominated by the National Academy of 
Sciences; 

• outstanding school-based professional educators; and 
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• individuals who are knowledgeable about educational needs— par- 
ents, chief state school officers, local education agency officials, 
principals, members of state or local boards of education or Bureau 
of Indian Affairs school boards, and individuals from business and 
industry. 

Ex officio members in addition to the Assistant Secretary for OERI include the 
directors of research for the Department of Defense and the Department of 
Labor; the directors of the National Science Foundation, the National Institutes 
of Health, the National Endowment for the Arts, and the National Endow- 
ment for the Humanities; the Librarian of Congress; and the director of the 
Office of Indian Education Programs at the Department of the Interior. 

It is now 4 years since the first meeting of the Board. During that time the 
Board has carried out its responsibility to approve standards for review of grant 
proposals, evaluation of exemplary and promising practices, and evaluation of 
OERI’s work. It has examined and made recommendations on the peer review 
system at OERI, crucial to ensure high-quality work. It has collaborated with 
the former Assistant Secretary to set initial priorities. It has reviewed solicita- 
tions for regional educational laboratories, the research and development cen- 
ters, and other major initiatives. The Board members have also stepped back to 
evaluate their functions more broadly, both to determine the effects of what 
members have done and to assess directions for education research in the future, 
especially as the time comes for reconsideration of the authorization of OERI’s 
research and development authorities. The following pages describe the findings 
and conclusions from these studies (section II) as background for Board goals 
and recommendations (section 11^. 
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II. The Board's Findings 

The Board has conducted its efforts with a view of education research, develop- 
ment, and dissemination as bringing opportunity and promise to learning for 
America’s youth. As the Board has surveyed the field of education research, it 
has tried to identify the major issues in the existing system— the longstanding 
problems and the increasing need for better returns from the investments in the 
system. It has grouped these in terms of resources, balance and linkage across 
the system, and processes. 

Resources 

Longstanding problems in education research start with insufficient resources. 
Limited funds have been spread thinly over a large number of topics and prob- 
lems, rather than concentrated on fewer issues. OERI’s national research insti- 
tutes, created in 1994 to re-focus education R&D on important educational 
topics and problems, are a prime example. Notwithstanding some bright spots, 
the institutes lack sufficient internal staff to mount credible programs consistent 
with their mandates for comprehensive and high-quality work and to provide 
national leadership on critical issues. This means that the “critical mass” found 
in other research institutions to be necessary for an effectivej high-quality 
program is missing. The Board’s concern about critical mass extends to the 
national educational research and development centers, which in many in- 
stances have too few resources for the work and leadership expected of them. 
The regional laboratories, as well, have immense formal missions, but only 
modest resources to achieve them. Some of these institutions have addressed the 
critical mass problem by aggressive efforts to obtain resources from other 
sources, but they all still face a mismatch between ambitious missions and 
limited resources to meet those expectations. 

If the quality, utility, and resources for education research are to improve, more 
effort, focus, and resources will be needed to strengthen the supporting infra- 
structure in three respects: 

(1) The “demand” for research should be strengthened. Building effective 
demand will require a substantial effort to educate teachers about the 
value and use of education research. That effort will succeed, however, 
only if educators are participants in the planning and execution of 
research. 
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(2) More resources can be used to attract new, highly qualified scholars 
from many disciplines to educational research and its issues. Lack of 
resources and prestige now inhibit such recruiting. 

(3) The institutions that undertake the critical work will need to be selected 
on merit, nurtured with sustaining resources and the demand for qual- 
ity, and rigorously evaluated for performance. 

The National Educational Research Policy and Priorities Board is not alone in 
its conclusion that education research is shamefully underfinanced. In 1997, the 
Panel on Educational Technology of the President’s Committee of Advisors on 
Science and Technology (PCAST) made several recommendations to improve 
the quality of education in all subject areas. The report did not focus only on 
technology as a topic of study, but as a means to strengthen content and peda- 
gogy in education, enhance professional development, and increase student 
learning. The Panel called for a large-scale program of rigorous, systematic 
research on education in general and educational technology in particular. It 
recommended an investment equal to at least 0.5 percent of our nation’s expen- 
ditures for elementary and secondary education— about $1.5 billion annually — a 
five-fold increase over what the Panel identified as the current level. That figure 
was contrasted with the pharmaceutical industry’s investment in research of an 
amount equal to 23 percent of all U.S. expenditures for prescription and non- 
prescription medications. 

Balance and Linkage 

The involvement of teachers and other education professionals in knowledge- 
building and implementation activities is stimulating new thinking about the 
design and conduct of research in education. It is increasingly clear that teacher 
acceptance of and success in revised practice is strengthened by understanding 
and involvement. This realization is leading toward efforts to seek active par- 
ticipation of teachers, schools, and districts in the research and development 
planning, and conduct and evaluation process. Some refer to this as creating 
“learning organizations.” In this role, the education professional community 
becomes vested in the objectives of the innovation and reform, provides helpful 
input in fitting concept with operational reality, and contributes a continuing 
basis for accountability and mid-course correction. 

The span of activities authorized for OERI is very broad, from fundamental 
research through large-scale demonstration and effective communication of 
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knowledge and information to the practitioner community. But that very 
breadth carries consequences when the research Investments that are needed to 
Improve student learning are not congruent with task. The portfolio of the 
Department Is aggregated In two areas: applied research and small-scale develop- 
ment; and communication activities. The Department conducts essentially no 
basic research, and is not deeply Involved in large-scale development or demon- 
stration, especially about comprehensive or standards-based reform. The cumu- 
lative science base supporting the applied agenda of the OERI and departmental 
R&D activities lacks clear visibility, which adversely affects Its force and cred- 
ibility. The OERI Institutes are contributing some of the Important applied 
research aimed at comprehensive or standards-based reform, testing and assess- 
ment. The Department’s participation in large-scale development has been 
generally modest. The pattern of foundation funding often mirrors the federal 
focus, although a few foundations are supporting large-scale comprehensive 
reform experimentation. Generally, foundations appear to give more focus to 
curriculum and teaching topics than the federal programs. It Is the Board’s 
Impression that the fit and relationship between foundation and federal funding 
are more happenstance than Intentional. The absence of substantial large-scale 
development activity aimed at critical problems with rigorous research and 
evaluation is noticeable. This concern Is particularly strong In light of the 
continuing difficulties of scaling up small, promising developments that require 
systemic change for widespread success. Providing knowledge of effectiveness at 
a large scale is an important Insulation against faddism and Insufficiently tested 
Ideas. 

Turning to dissemination, OERI’s work most frequently has followed tradi- 
tional approaches, no longer believed adequate, which leave to those In need of 
exemplary practice and sound knowledge the burden of finding It. This Is 
particularly true for those undertaking large-scale comprehensive reforms. Even 
with the Internet and other forms of user-friendly electronic access, the passive 
systems do not fully meet the needs of those with ambitious Innovation agen- 
das, and the volume of Information can overwhelm the practitioners Involved. 
The more Intensive efforts appear to require a combination of traditional 
dissemination, technical assistance, and short-term applied research or problem 
solving. A new set of Intermediaries and adjustments In existing organizations 
are emerging to meet these needs, and dissemination must be reconceptualized 
In this broader context. 

The Board has learned that fundamental research-^largely In the cognitive and 
neural sciences — Is conducted In other federal agencies, most notably the Na- 
tional Institute of Child Health and Human Development (NICHD), the 
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office of Naval Research, and to a lesser degree the National Science Founda- 
tion (NSF). There is also modest foundation support for basic educational 
research. The location and level of fundamental research is of concern in several 
different ways. It is not desirable that basic research be sponsored or conducted 
in one organizational framework, but it is important that such work be linked 
to the applied research and ultimate practitioner communities that will exploit 
and make use of its findings. Sound linkage requires that staff in applied re- 
search organizations are sensitive to implications of the findings and are quali- 
fied to design the applied and related basic research needed to push sound 
findings toward utilization. Such linkage requires continuing identification of 
application problems and unmet needs from practitioner and applied research 
communities. Efforts to link across organizational performers are occurring 
more frequently, such as recent planning work concerning a new initiative 
among OERI, NICHD, and NSF. 

The research planning processes in Education are newer and less well articu- 
lated than those for Defense and Health. Department plans set forth broad goals 
and objectives, and do a particularly commendable job of relating research and 
development efforts to mission objectives. But they are unlike the Defense and 
Health counterparts in two undesirable ways. First, they are much clearer 
about the mission and application goals than about the research goals and 
priorities. Second, there is a strong sense in the defense and health cases that the 
science base is firmly rooted, and that there is a clear sense of direction and 
cumulative learning. Moreover, in those cases, the growing knowledge base is a 
powerful determinant of both future research and operational actions. There is 
no such comprehensive sense for education research and development. These 
differences are in part attributable to the huge difference in resources among the 
three agencies, which inevitably affects the style that has been adopted. But 
more than just staff and dollar resources are at issue; there is also the question of 
whether the education research and development program is an endless series of 
small applied research projects unrelated to an evolving critical set of knowl- 
edge bases or a cohesive agenda of cumulative knowledge-building. By this 
criterion, these other federal agencies are better developed. 

Processes 

The Board’s review of the current education research system included specific 
attention to important processes by which agendas are set, support is mobilized, 
resources are allocated, and progress is achieved, assessed, and made known. 

The experiences and models in other federal R&D programs provide insights 
for assessment of OERI’s work. 
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• Balance of researcher initiatives with national focus-. The first area is 
agenda setting, in which two important objectives need to be harmo- 
nized. On the one hand, long experience suggests that R&D is most 
productive if researchers are given substantial latitude in initiating 
work on their own ideas or perceived problems, generally known as 
field-initiated research. On the other hand, there is a strong and 
continuing need to give the R&D agenda a sense of national focus 
and priorities aimed at the most important gaps in our knowledge 
and the most promising research approaches. OERI has adopted 
several devices to create the desired harmony, yet the national focus 
part of the balance remains weaker than it should be. Education 
lacks a process similar to that employed in NIH, for example, that 
could ensure a continuing dialog among OERI, the Institutes, Cen- 
ters and Regional Laboratories plus other important education 
research institutions and sponsors about critical research problems 
and opportunities. 

• Respect for research as well as for policy decisionmaking-. A second issue 
centers on mobilization of support. Given the importance of federal 
funds in education R&D, garnering support means the involvement 
of the federal political process. Respect for the appropriate domain 
and responsibilities of the research community and political policy- 
makers is both necessary and is accomplished in other federal R&D 
programs. Where it works well, all parties are engaged in setting 
broad objectives and parameters of the work, while leaving specific 
design and execution of the projects to the sponsoring agency and 
the research and practitioner communities. 

• Meeting high standards of quality: The third issue is development of 
appropriate research methodologies and the establishment of stan- 
dards of evidence to be applied to knowledge-building activities. 
More rigor is clearly desired and needed in education research. 
Traditional rigorous methodologies such as randomized experiments 
may not routinely be feasible, affordable, or appropriate in education 
research, though control group methodology has a continuing and 
important role in the clarification of uncertainty about critical 
research issues. Members of the Board believe that further efforts are 
needed to define appropriate research designs. The choice is not 
between randomized techniques and nothing at all. There is an 
appropriate level of rigor associated with the stage and purpose of 
the research being undertaken. 
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III. The Board’s Goals and 
Recommendations for Educational 
Research 



In the 1994 “Educational Research, Development, Dissemination, and Improve- 
ment Act,” Congress set forth a powerful challenge for education research and 
for the National Educational Research Policy and Priorities Board: 

The Congress declares it to be the policy of the United States to provide 
to every individual an equal opportunity to receive an education of high 
quality ... To achieve (that) goal . . . requires the continued pursuit of 
knowledge about education through research, development, improve- 
ment activities, data collection, synthesis, technical assistance, and 
information dissemination. While the direction of American education 
remains primarily the responsibility of State and local governments, the 
Federal Government has a clear responsibility to provide leadership in 
the conduct and support of scientific inquiry into the educational pro- 
cess . . . The failure of the Federal Government to adequately invest in 
educational research and development has denied the United States a 
sound foundation of knowledge on which to design school improve- 
ments . . . (The) National Educational Research Policy and Priorities 
Board should . . . work collaboratively with the Assistant Secretary to 
forge a national consensus with respect to a long-term agenda for educa-. 
tional research, development, dissemination, and the activities of the 
Office. 

As a result of its studies, and its meetings with teachers, educators, researchers, 
policymakers and others, the Board has reached consensus on four goals that are 
critical to meet the challenges expressed by Congress in the 1994 legislation. 
These goals are statements about characteristics of research in education. If the 
goals were reached, there would be a sound basis for trust in the results and 
growing support for conduct of research. The Board’s goals are: 

• Priorities are set and activities are problem-centered. 

• High standards of quality are created and upheld. 
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• Work is collaborative and rigorous. 

• Mission is congruent with resources. 

The sections below provide additional information about each of these goals 
and make recommendations for action to achieve them. 

Priorities Are Set and Activities Are Problem- 
Centered 

In the Board’s work with the Assistant Secretary to establish a priorities plan, 
and its subsequent efforts to refine and target priorities, it has found that a 
problem-centered focus for research for developing research agendas works best. 
That is, identifying real problems faced by teachers in real classrooms is the 
most understandable way to design and target appropriate research. The princi- 
pal priority should be teaching and learning and, more specifically, improved 
achievement for all students. But balanced research agenda setting must also 
give weight to identification of research opportunities— where research is poised 
for advances. The targets for action should be ones for which there is reason to 
be optimistic that research has something important to say, or could have, with 
the proper investment. 

Together, the Board and OERI sponsored a 1998 conference on “National 
Directions in Education Research Planning' that brought together leaders and 
representatives from a dozen or more research-planning efforts under way 
among federal agencies, professional and scientific organizations. Its purpose 
was to put individuals associated with those efforts into communication with 
one another and with the educators and policymakers who could use the fruits 
of education research to enhance learning and suggest appropriate priorities and 
collaborations for current and future work. 

The overriding sense of the conference was that educational research planning 
must emphasize focus and selectivity. The Board heard a consensus among 
conference participants that education research should be concentrating its 
inquiries on those areas that the public and the profession believe are impor- 
tant, as well as anticipating problems that will become important. Among the 
conferees, the appropriate topics were identified as reading and language learn- 
ing; expanded attention to mathematics; the dynamics of teacher performance 
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and effectiveness in schools and classrooms; and new emphasis on technology 
and telecommunications; international studies; and learning in family, com- 
munity, and workplace settings. 

Student Achievement 

Members of the Board believe that the focus of research attention must be 
narrower still, in order to concentrate on something both important and 
possible. 



Recommendation 1: Student Achievement— The priority for 
research in education must be high achievement for all students 
and, within that domain, the initial emphasis should be on 
reading and mathematics achievement. 

Raising student achievement is a priority for education supported alike by 
parents, business leaders, public officials, and educators. But to attain high 
achievement for all students requires success in combating the most difficult 
and challenging issues of student performance across America. These are 
issues sharpened for us once again, recently, by international comparisons 
from the TIMSS in which both the strengths and the shortcomings in 
achievement among our youth are apparent. TIMSS data indicate that our 
younger children, age 9, demonstrate mathematics and science knowledge 
and skill at levels approximating those of children in other economically 
developed countries. But as they progress through the school system, they 
fall farther behind, so that by 12th grade, American students are among the 
lowest scoring students in the study. Before the TIMSS results were released, 
we may have been able to take comfort that our most able students ranked 
with those of other nations, but that has now been disproved as well. The 
TIMSS results repeated findings from the National Assessment of Educa- 
tional Progress (NAEP) and other sources, showing that achievement of 
children from minority backgrounds and from low-income families, on 
average, continues to lag far behind that of the majority population. 

These characteristics of student achievement — that (1) it is below levels 
experts believe necessary for maintenance and preservation of American 
democracy and for full participation in a vibrant economy in the 21st cen- 
tury, and (2) that there are unacceptably wide gaps across members of our 
population — are longstanding and have resisted well-intentioned attempts 
over many years to remedy. Members of the National Educational Research 
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Policy and Priorities Board believe that the combined efforts from researchers 
across several disciplines, developers, and teachers can make a difference If 
student achievement Is the priority. 

We have called on the National Academy of Education (the Academy) to 
provide assistance In forming a research agenda around high student achieve- 
ment, one that would build on what Is already known, and one that would 
capture the most promising areas for further exploration. The Academy was 
asked to create a possible agenda on a series of focused research questions. The 
Academy has suggested three strands of work. The first strand Is research on 
learning, especially across transitions In children’s lives; the second on teaching 
in relation to learning as a professional practice to support student learning; and 
the third, strengthening the links between educational research and the practice of 
schooling. The first two are discussed In this section on priorities and problem- 
centered activities. The third Is discussed below under collaborative work 
because It Is directed toward constructing more powerful methodologies for 
conduct of research In education— not only In the area of high achievement for 
all students, but other topics as well. 

The ability of the United States to make substantial progress toward the goal of 
high achievement for all students Is limited by assumptions about the nature of 
research, learning, and teaching that cause policymakers and practitioners to 
neglect Important complexities associated with education. It Is, for example, 
usually assumed that the results of researchers’ Investigations should have 
Important practical Implications, whether or not the researchers are trying to 
Improve educational effectiveness. Regarding students’ learning, students are 
usually assumed to learn procedures and facts Independently of their compre- 
hension of the concepts and principles that make them understandable, and 
students’ learning in school is assumed to occur independently of their abilities 
and personal Identities outside of school. Teachers are assumed to develop skills 
and subject-matter knowledge independently of the social and cognitive chal- 
lenges they deal with in their classrooms as they interact with students. In fact, 
however, the relations among all these factors must be better understood, and 
to do so, education must be thought of as a complex professional undertaking. 

Transitions 

In the first strand of its measures for focusing research on high achievement for 
all students, the Academy has advised us that the most critical and promising 
research questions fall into two areas. One involves transitions that students 
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must accomplish in order to progress successfully through the school curricu- 
lum. The other concerns transitions that involve the social organization of 
learning in schools and its relation to the activities of students outside of school. 

The Board recommends that research on teaching and learning in school be 
focused on critical transitions that include important developments in concep- 
tual understanding as students move through the school curriculum, and as they 
move between the school and other communities. To achieve at high levels, 
students must succeed in critical transitions that require mastery in general 
aspects of knowing and understanding that are often not explicitly taught. The 
expectations in school for these general aspects of understanding and learning 
do not match with the experiences of all students, and the transitions are much 
easier for students for whom the school routines and practices are in close 
alignment with those that prevail at home. The difference is generally unfavor- 
able to students of low-income families. 

Recommendation 2: Reading, second language learning, and 
mathematics — ^Recent reports from the National Research Coun- 
cil, Preventing Reading Difficulties in Young Children, and Im- 
proving Schooling for Language-Minority children: A Research 
Agenda, synthesize strong bodies of research knowledge. A 
similar study on mathematics is currently under way. In each 
case, research is needed now to analyze how the results of our 
knowledge can be implemented in school programs and what 
factors lead to success and difficulties. In reading, research is 
needed on how students become facile at reading complex text as 
they transition to advanced academic subjects such as history, 
social science, mathematics, and science. Research in both short- 
and long-term effects of specific education interventions for 
English language learners is needed, as well as techniques of 
assessment to measure competence, and transition points (a) from 
the first oral language to English, (b) from oral language to 
literacy, and (c) from literacy to the academic discourse of spe- 
cific disciplines. In mathematics, research is needed on why 
students have so much trouble making transitions (e.g., from 
concrete objects to abstract ideas), understanding formal repre- 
sentations, multiplicative reasoning, and essential mathematical 
and statistical concepts such as chance, randomness, and probabil- 
ity. 

High achievement for all students will not be accomplished by policies and 
practices that consider only students’ activities in school without taking account 
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of the competencies that students, especially students from backgrounds of 
poverty, develop in other aspects of their lives. Several recommendations focus 
on ways in which the social arrangements of learning in school, as well as the 
content of school learning activities, need to be studied and understood to 
inform policies and practices that can lead to high achievement for all students. 

Recommendation 3: Organization for learning out-of school — ^To 
take advantage of learning environments in which children from 
impoverished backgrounds often display more competence than 
in school settings, research is needed to design and test different 
models of after-school and summer programs to motivate, en- 
gage, and benefit low-income children. Work is also needed on 
types and features of after-school opportunities that most effec- 
tively motivate academic achievement and positive self-estima- 
tions; and how to design and test different models of collabora- 
tion between schools and community groups dedicated to pro- 
viding strong learning environments for disadvantaged children. 

Recommendation 4: Organization for learning in-school — Reten- 
tion, pull-out remediation, tracking, and segregated special 
education programs that stratify by race, class, and gender oppor- 
tunities to learn do not result in high achievement for all stu- 
dents. A more complete inventory of knowledge about effective 
practices for teaching academically challenging curricula with 
groups is needed, both for school populations in general, and for 
heterogeneous groups in particular. Research is needed on ques- 
tions of time for children to master challenging curricula, sup- 
portive school structures, and expectations for the breadth and 
depth of content. Within each of these is the question, do stu- 
dents from middle-income families as well as students from low- 
income, ethnic, and linguistic minority backgrounds benefit 
from each organizational practice? An important area of inquiry 
is whether there are academic benefits to classroom diversity- 
does diversity improve subject-matter learning? (Note: Recom- 
mendation #5 addresses teaching practice aspects of school orga- 
nization for learning.) 

Teaching and Learning 

The second strand of research to promote high student achievement in reading 
and mathematics is teaching as a professional practice. Without improving our 
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understanding of what it will take to produce a well-prepared and professional 
corps of teachers, school improvement will not be possible. Students living in 
poverty and ethnic minorities have been historically underserved by American 
educational institutions and are an increasingly large proportion of the student 
population. No one doubts that teachers will have much to learn in the years to 
come in order to be successful in helping all students reach high levels of 
achievement. There should be a particular concern with producing new knowl- 
edge about connections between professional development and improving 
education for currently underserved populations; namely, children and adoles- 
cents whose experiences and dispositions do not match with the expectations 
and social organization of schools. 

Recommendation 3\ Linking changes in teaching practice with 
improved student learning — ^Information is needed that can guide 
teachers and institutions who want to change their educational 
practice, particularly to reduce inequities in the opportunities of 
students who differ in socioeconomic status, ethnic background, 
and gender to learn successfully. This is especially important 
regarding the achievement of deep intellective competence advo- 
cated in current educational reforms. Such research would exam- 
ine fundamental issues about the nature of teaching and learning, 
including, but not limited to, the importance of the skills and 
knowledge of teachers. Expansion is needed in our knowledge 
and understanding of teaching practices, including teaching tools 
such as assessment, that are successful with students who bring 
different cultural resources to their own and to other students’ 
learning. Research would examine, much more than past research 
has done, issues of what it takes to do effective and successful 
teaching with diverse populations of students. (Note: Recom- 
mendation #4 addresses the school organization context in which 
effective teaching for student learning takes place.) 

Recommendation 6: Linking teachers’ professional development and 
teaching practices — Research is needed to understand what effec- 
tive teachers do and how they do it. Successful teaching involves 
not only the exercise of skills and application of knowledge but 
also flexible improvisational adaptation in classroom circum- 
stances. Research is needed to understand the roles of more 
profound knowledge and comprehension of subject-matter 
concepts and methods, both at the level one is teaching and in 
relation to other disciplines and grade levels, as well as the role of 
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understanding processes of students’ learning. Research should 
also investigate how the structure of teachers’ work supports or 
hinders their “on-the-job” learning and what kinds of abilities are 
learned in particular situations that can transfer to other settings 
with different circumstances. 

Recommendation 7: Understanding and supporting successful 
professional development — ^There is need for a better understand- 
ing of teachers’ development as “professional learning,” and 
understanding teaching as a “learning profession.” The prevalent 
model of learning how to teach — the knowledge goes in during 
teacher education or professional development and then comes 
out to be used in the teachers’ own classrooms— does not account 
for the engagement of teachers themselves in improving the 
practice of their profession. What teachers need to learn to put 
reforms in place is not separable from their actual teaching 
practices or from the development trajectories of their careers. 
Research must examine ways in which people of diverse cultural, 
ethnic, and socioeconomic backgrounds are attracted to careers 
in teaching and how professional development resources can help 
increase and maintain diversity in the teaching profession, while 
continuing to improve practice. Further research on teachers’ 
communities of practice is needed, building on findings that 
norms of responsibility and collegial efforts at professional 
problem solving are the most critical factors in improvement of 
teaching and learning. 



Assessment 

Focusing the research agenda as the Board has suggested means that some 
important topics will not receive much attention. For example, such areas as 
cultural and political contexts of schools, educational policy, and school fi- 
nance; education governance; and learning environments and educational 
technology— all of them areas in which important work might be done if 
sufficient resources become available — would not receive significant attention 
under the Board’s view of priorities. 

Among all the topics that would be deferred under the Board’s identification of 
priorities to achieve high achievement for all students, one that the Board 
would single out as a candidate for inclusion at the earliest opportunity is assess- 
ment of teaching and learning for purposes of accountability. 
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Accountability is an increasingly important issue in educational research and 
practice. Current accountability measures, however, do not match the goals of 
most educational reforms for students of low-income families, especially those 
reforms aimed at improving their complex thinking and participation in activi- 
ties of inquiry and understanding. One issue to be considered is the limitations 
of norm-referenced tests that are conventionally presented in standardized, 
multiple-choice formats. Criterion-referenced measures, aligned with teaching 
and learning standards, may assess the competence of these students more 
productively than norm-referenced tests. Alternative methods that are respon- 
sive and valid guides for instruction of students who come from a background 
of poverty should be developed and studied as soon as adequate funding can be 
attained. 

High Standards of Quality Are Created and 
Upheld 

The single criterion by which any scientific enterprise must be judged is the 
quality of its work. Scientific norms must be known and shared. The expecta- 
tions for explicit hypotheses, sound designs, appropriate measures, sufficient 
data of good quality, and logical analyses must be widely shared. High standards 
must be insisted upon in all areas of a scientific agency’s work-in selection of 
proposals, design of appropriate methodologies, creation of research agendas, 
identification of effective and promising practices, and evaluation of all efforts it 
conducts or supports. 

The primary means by which high standards have been developed and assured 
in federal agencies has been through extensive networks to assure involvement 
of peers. Peer review is much more than a bureaucratic instrument. It is a major 
vehicle of communication between the government and the field, a process 
through which principles about fesearch priorities and technical quality of 
research are clearly articulated, and applied to proposals. In the 1994 legislation. 
Congress made its intent clear that a peer system must be applied to every 
aspect of OERI’s work. The law requires: 

that a system of peer review be utilized by the Office — for reviewing and 
evaluating all applications . . . which exceed $1 million; . . . evaluating 
and assessing the performance of all recipients of grants; . . . cooperative 
agreements and contracts; . . . and for reviewing and designating exem- 
plary and promising programs . . . 
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In addition, the law requires OERI to adopt, and the Board to approve: 

such standards as may be necessary to govern the conduct and evaluation 
of all research, development, and dissemination activities carried out by 
the Office to assure that such activities meet the highest standards of 
professional excellence. In developing such standards, the Assistant 
Secretary shall review the procedures utilized by the NIH, NSF, and 
other federal departments or agencies engaged in research and develop- 
ment and shall actively solicit recommendations from research organiza- 
tions and members of the general public. 

While the work of the Board since 1995 has frequently centered on the prepara- 
tion and approval of those standards, the Board has also undertaken a review of 
the set that has been in place longest— standards for approval of grants— to 
determine (a) whether the standards are appropriate and useful; (b) whether 
they contribute to fair and high quality competitions; (c) how the process 
worked and how it might be improved; and (d) what recommendations might 
be made on how to configure and maintain peer review panels. 

In this assessment of operation of the standards in fiscal years 1996 and 1997 and 
in two types of competitions — field-initiated studies and Research Centers — 
Board members learned that as many as a third of reviewers had not conducted 
research in education, even though that is a requirement in the standards. 
Among those who might have had research training and had themselves con- 
ducted research, that training and research experience was in broad topical areas 
related to the competition, but not necessarily in the methods and design of 
research in the proposals. In examining the reviews provided by OERI panel- 
ists, the Board-commissioned study found that most reviews provided little 
depth in their commentaries. Reviews were most detailed about project design 
and significance, least detailed on staffing, budget, and management plans. 
Applicants frequently disagreed with reviewer comments, saw the comments as 
superficial or irrelevant, found a lack of comments about design, and cited a 
lack of examples. Applicants also noted limited explanations and mentioned 
large discrepancies among reviewers. They believed that proposals had not been 
carefully read and said that comments were illegible. 

Standing Panels 

As a result of this review, the Board’s principal recommendation on research 
quality is addressed to the establishment of standing panels. 
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Recommendation 8: Standing panels — ^Standing panels should be 
established to review proposals for each OERI Institute. These 
would be comprised of 25 to 30 members, but with some over- 
lapping membership, so that problems that cross boundaries can 
receive informed attention and so that members of one panel 
with special knowledge could be invited to serve on another 
Institute panel. 

This is the Board’s most urgent and important recommendation on peer review 
processes, and it can be implemented by OERI on its own authority both easily 
and relatively quickly. Standing panels, frequently used by such organizations 
as NIH, offer the most compelling mechanism the Board could find to improve 
the quality of the review process. Standing panels provide continuity from the 
announcement of government funding opportunities to the decisions on pro- 
posals to fund. They provide an informed group to build areas of research over 
time so that the results are cumulative rather than episodic. Such panels can 
attract experienced members who will agree to serve, because the repeated 
contacts with colleagues are more professionally rewarding than membership in 
ad hoc settings. They can provide a forum where the accumulating knowledge 
can be sifted and interpreted, and new lines of research can be identified. 

Thus, standing panels are a device to attract the very people whose judgment is 
needed to ensure that research proposals are of the highest scientific merit and 
are addressed to high priority national education needs. They can also play a 
crucial role in guiding and evaluating the direction of research. 

The Board has additional recommendations that complement and support the 
standing panels. 

Recommendation 9: Panel membership — Panels should represent a 
broad range of perspectives. They must include members with 
strong disciplinary and methodological expertise. Across OERI’s 
panels, gender, race, ethnicity, and geographic diversity must be 
respected. Panel members should be nationally recogni 2 :ed fig- 
ures. 

Recommendation 10: Standards for panelists— Board continues 
to support peer review standards adopted by OERI, with Board 
approval, which specify that all reviewers meet three criteria: “(a) 
demonstrated expertise, including training and experience, in the 
subject area of the competition; (b) in-depth knowledge of policy 
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or practice in education; and (c) in-depth knowledge of theoreti- 
cal perspectives or methodological approaches in the subject area 
of the competition.” 

Enhancing Quality in Competitions 

The Board has made further recommendations to the Assistant Secretary to 
enhance the effectiveness of reviewers, reduce workload for reviewers and 
applicants, and improve center competitions. For all grant competitions, it is 
important that the quality of research designs be ra;ted by reviewers with appro- 
priate technical expertise. The Board strongly prefers that each proposal be read 
by a minimum of five people. More logistical and other support should be 
provided for reviewers along with more in-depth training, and better formats 
should be created in the technical review form to guide the reviews. Reviewers 
should be expected to provide specific, but not necessarily detailed, feedback to 
applicants. Reducing workload for both reviewers and applicants would en- 
hance the likelihood of accomplishing these changes. For example, the Assistant 
Secretary should consider making grant announcements and appointing submis- 
sion dates earlier in the fiscal year; reducing the number of full applications 
through use of preliminary applications; reducing the number of pages permit- 
ted for center applications and the page limit for attachments; assignment of 
specific primary, secondary, and tertiary reviewers to applications; and conduct 
a small pilot project to determine how technology might be used to support the 
peer review process. 

For center competitions specifically, the Board has urged the Assistant Secre- 
tary to clarify the project design criterion so that reviews address the end 
projects proposed as well as the overall center design; increase weighting for 
management and clearer instructions; and provide planning grants. These 
changes require modifications in regulations. 

The Board also has recommendations for enhancing peer review processes that 
may require changes in the 1994 Act. 

Recommendation 11: Distinguishing between field-initiated and 
directed competitions — OERI should distinguish between field- 
initiated competitions and those that are directed, rather than 
trying to combine the two. 
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Recommendation 12\ Funding for peer review — ^The allowable 
percent of funds to support peer review should be increased so 
that the necessary standing panels may be established and 
logistical support provided. 

Recommendation 13: Definitions — ^The term “research” should 
have a narrower definition than it has in the 1994 law so that 
the boundaries of focused competitions for research can be 
limited. Research should encompass basic research in educa- 
tion as well as investigations, experiments, and inquiry to 
develop new knowledge or apply tested knowledge. It should 
exclude development, planning, and demonstrations. The 
term “national significance” needs to be clarified through 
regulations or in legislative language so that reviewers under- 
stand that it includes research opportunities, not only impor- 
tant problems identified by educators. 

Work is Collaborative and Rigorous 

Collaborative Research 

As noted in section I, the 1994 Act sets the tone for collaboration in all of 
OERI’s work, including that of the Board in its relationships with the Office 
and the Assistant Secretary. OERI should conduct all its work in ways that 
bring diverse perspectives constructively together. This includes the perspec- 
tives of researchers, educators, policymakers, the public; representatives of 
the nation’s diverse populations and cultures; federal agencies participating in 
the conduct of education research, as well as states, foundations, and the 
private sector. To the extent appropriate for each function, this range of 
perspectives should be represented in all of OERI’s activities, from develop- 
ing agendas, to selection of awardees, oversight, evaluation, and refinement. 

OERI has an important place in research in education throughout the De- 
partment and across the government. This is defined by Congress in broad 
terms in the 1994 legislation. For example, the law sets forth a coordination 
role for the Assistant Secretary and OERI; 

With the advice and assistance of the Board, the Assistant Secretary 
shall work cooperatively with the Secretary and the other Assistant 
Secretaries of the Department of Education to establish and maintain 
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an ongoing program of activities designed to improve the coordination 
of education research, development, and dissemination and activities 
within such Department and within the Federal Government. 

The law goes on to specify the goals of minimizing duplication, maximizing the 
value of federal investment, and enabling entities in education research to 
interact effectively as partners. 

But OERI is limited in its abilities to achieve this role. It provides, for example, 
only a small part of the total support for education research and development. 

A study prepared for the Board estimated U.S. spending for research in educa- 
tion in the range of $900 million to $1 billion through the U.S. Department of 
Education and among foundations. A larger net that includes investments in 
education studies and data collection in other federal agencies such as the De- 
partment of Defense, the Department of Health and Human Services, and the 
National Science Foundation, as well as state and local governments and univer- 
sities might add another billion. With a $2 billion level of annual expenditures 
for education research activities, the nation would be investing less than 0.5 
percent of the total enterprise in educational knowledge-building. However, the 
OERI share represents only about one-fifth of that of the Department and 
about one-twentieth of the $2 billion estimated national education research 
investment. 

Aside from the level of investment is the question of OERI’s span of activities 
in education research. To be blunt, OERI is only one among several agencies 
involved in significant support and or conduct of education research. Within 
the Department, research related to individuals with disabilities and education 
of children with disabilities is considerably larger. The Department of Health 
and Human Services conducts research on learning, family structure, integrated 
service delivery, and funds dissemination activities related to education. The 
Department of Labor funds research on dropouts and illiteracy, and funds 
dissemination activities related to education. The National Endowment for the 
Humanities funds research and dissemination on students’ knowledge of history 
and the humanities. The National Science Foundation has worked on the 
teaching of math and science, NIH in learning disabilities and reading, and the 
Office of Naval Research (ONR) in cognitive and neural science and technol- 
ogy. 

These simple facts have impressed upon the Board that the almost unbounded 
role envisioned for OERI in the 1994 legislation creates unachievable ends. 
OERI is not a monopoly, not the most significant element, not the leading 
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federal influence in several prominent substantive areas. The reality is that 
OERI must carefully balance its own initiatives and its collaboration with other 
federal agencies, foundations, and states and localities. 

In the coordination of Department research and in exercise of its authorities for 
government-wide collaboration, OERI and the Department, nevertheless, have 
vital strengths that all the government agencies can respect, in that they: 

• retain substantial credibility and utility as convenors — they can bring 
people together; 

• maintain unparalleled connections with the nation’s educators and 
education policymakers; and 

• support the policy goal of assuring that the nation’s education resources 
are used effectively to provide opportunities to those individuals in our 
population who have traditionally not been served well. 

OERI’s strengths can often be used particularly well in combination with 
complementary attributes of other agencies. The Board makes the following 
recommendations simply to reinforce the letter and spirit of the 1994 OERI 
law: 

Recommendation 14: Collaboration across federal agencies — ^The 
Assistant Secretary should extend efforts to join with other 
federal agencies, and perhaps foundations, to collaborate on 
common agendas. 

Recommendation 15: Coordination of research within the Depart- 
ment of Education — The Secretary should encourage, and the 
Assistant Secretary should provide, special attention to perform- 
ing a visible and constructive role in collaboration and coordina- 
tion of education research within the Department of Education. 

The Board does not view these as mere bureaucratic exercises. Every effort 
should be made to create constructive tasks for which it is to the advantage of 
all collaborators to join with the Assistant Secretary. Some examples of what 
the Board members expect might include (1) large areas of research that a single 
agency would not have the resources to undertake alone, (2) syntheses across 
important topics such as development or learning in which the research sup- 
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Recommendation 22: Research supervision — Some focal point 
should be created by Congress for research leadership that can 
span across administrations. 

In addition to generally insufficient funding, the allocation of resources for 
OERI, particularly to and by the institutes, is heavily controlled by statutory 
and other distribution rules. These rules frustrate responsiveness to new needs 
and circumstances, fractionate limited funds, and inhibit response to new 
national priorities. In a well functioning system, these rules should not be 
necessary. 



Recommendation 23: Allocation requirements- — ^The allocation 
provisions in the 1994 Act for institutes and for types of support 
should be removed. 
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IV. Concluding Observations 

To supplement its recommendations for research in education, the Board has 
some concluding observations about its own work, and particularly about how 
it can best be effective. The legislation that created the Board contains numer- 
ous references to “collaboration.” In addition, the Board is required by law to 
“review regularly, evaluate and publicly comment upon” actions of the adminis- 
tration and Congress. Among provisions of the law is one that says the Board 
must be offered an opportunity to provide written comments on any proposed 
“grants, contract, or cooperative agreements” over $1 million, and those com- 
ments may cover both consistency of the proposed use of funds with the re- 
search priorities and the “methodology and approach of the proposed actions.” 

Taken together, these are potentially powerful authorities, although they must 
be exercised in balance with the resources the Board has available or can attain. 
On one occasion when appropriations exceeded the administration’s budget 
request, the Board was invited to provide an overall analysis and make recom- 
mendations prior to Department decisions about use of these appropriations. 
The Board believes that was an especially effective interchange between the 
administration and the Board and one that permitted the Board to be both 
constructive in its comments and effective in its influence over the subsequent 
actions. However, there have been other occasions, such as preparation of the 
President’s budget and the administration’s proposals for a national voluntary 
test, on which the Board’s involvement came not only after the decisions were 
made but after public debate had begun. These latter examples do not seem to 
demonstrate the collaborative relationship that the law seeks to establish. Even 
worse, perhaps, they fail to take advantage of the counsel the Board was created 
to provide. It is just that structure and those processes that the Board has used 
to prepare this policy statement. This is one concern. 

Of greater importance, however, is the view of Board members that much has 
been accomplished during the past 4 years to serve as a platform for the future. 
The members have learned to work through the diversity of views that Con- 
gress wisely insisted be represented among the appointees. All have come to 
appreciate the potential of sound research in education as a means through 
which all American students can become better prepared for their lives in a new 
millennium. 
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Members of the National Educational Research Policy and Priorities Board 
appreciate the opportunity that Secretary Riley provided them through ap- 
pointment to the Board to serve American education. They are committed to 
perform, and eager to continue, their special functions in policy and priority 
setting for education research. 



44 



47 



Investing in Learning 



References and Works Consulted 



Publications: 

Elliott, Emerson. The National Educational Research Policy and Priorities Board, 
Its Role, Development, and Prospects. January 1999. 

U.S. Department of Education. National Educational Research Policy and 

Priorities Board. Creating a New Research Agenda on Race, Gender, and 
Class Impacts on Educational Achievement and Underachievement. Con- 
ference Proceedings. March 1998. 

U.S. Department of Education. National Educational Research Policy and 

Priorities Board. Attaining Excellence: A Handbook on the Standards for 
the Conduct and Evaluation of Research. April 1999. 

U. S. Department of Education. Office of Educational Research and Improve- 
ment and National Educational Research Policy and Priorities Board. 
Building Knowledge for a Nation of Learners: A Framework for Education 
Research, 1997. 

U.S. Department of Education. Office of Educational Research and Improve- 
ment and National Educational Research Policy and Priorities Board. 
National Directions in Education Research Planning. Conference Proceed- 
ings. June 1998. 

Vinovskis, Maris. Changing Federal Strategies for Supporting Educational Re- 
search, Development, and Statistics. September 1998. 

Commissioned Papers 

American Institutes for Research/Mathtech. A Comparison of Selected Federal 
RD&D Styles. March 1998. 

. The Educational Research, Development, and Dissemination System. March 

1998. 






Investing in Learning 



45 






ERiC 



References and Works Consulted 



. The Needs, Demand For, and Supply of Educational Research, Development, 

and Dissemination. February 1998. 

. A Review of the Federal Role and the Department of Education Structure. 

March 1998. 

. Shaping the Future of Educational Research, Development, and Communi- 
cation. October 1998. 

August and Associates. Strengthening the Standards: Recommendations for OERI 
Peer Review, Summary Report. January 1999 (prepublication copy). 

American Academy of Education. Recommendations Regarding Research Priori- 
ties. April 1999. 

Transcripts: 

The Current and Future RD&D System. NERPPB Workshop, 

September 25, 1997. 

Envisioning and Educational Research, Development, and Dissemination System. 
NERPPB Workshop, March 20, 1997. 

NERPPB Quarterly Meetings, March 18, 1999; September 18, 1998; June 18, 
1998; March 19, 1998; and January 16, 1998. 



46 






Investing in Learning 



Investing in Research: 

A Second Policy Statement 
with Further Recommendations 
for Research in Education 



O 

ERIC 



50 



Investing in Research: 

A Second Policy Statement 
with Further Recommendations 
for Research in Education 

In this second set of policy recommendations on research in education, the Na- 
tional Educational Research Policy and Priorities Board addresses 

• the urgency of increased rigor in education research, 

• several questions concerning the federal government’s organizational arrange- 
ments, and 

• the policy setting and leadership functions for research. ' 

The Board’s initial policy paper, Investing in Learning, contains recommendations on 
building an education research system in the federal government. It envisions the 
Office of Educational Research and Improvement (OERI) performing primary 
functions, but acknowledges the extensive roles performed elsewhere in the U.S. 
Department of Education and other agencies — such as the National Science 
Foundation, the National Institute of Child Health and Human Development in the 
U.S. Department of Health and Human Services, and the U.S. Department of 
Defense. The Board addresses four topics essential to the development of capacity 
within OERI: 

• structuring an agenda around a few critical problem-centered priorities; 

• creating and upholding high standards of quality; 

• conducting research in collaborative and rigorous fashion; and 

• aligning the mission and resources of OERI. 

Urgent as these topics are for OERI’s research mission, they do not fully address 
other issues particularly salient to the legislative reauthorization cycle in which 
OERI now finds itself. For that reason, the Board has prepared this second set of 
recommendations. 
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Recommendation 1: Designing research for credible results— Ke- 
search in education should be conducted consistent with rigorous 
standards, reflecting scientific principles and appropriate for the 
matters being investigated. Congress and the Department should 
require annual state-of-research reports describing progress toward 
incorporation of rigorous research designs in all OERI program 
portfolios. 

In its previous policy statement, the Board asserted that the single criterion by 
which any scientific enterprise must be judged is the quality of its work. Scientific 
norms must be known and shared. The expectations for explicit hypotheses, sound 
designs, appropriate measures, sufficient data of good quality, and logical analyses 
must be widely shared. High standards must be insisted upon in all areas of a 
scientific agency’s work-in selection of proposals, design of appropriate method- 
ologies, creation of research agendas, identification of effective and promising 
practices, and evaluation of all efforts it conducts or supports.^ The earlier state- 
ment also observes that education improvement occurs slowly and in small incre- 
ments no matter how powerful the research base behind it. Among other things, 
research requires reflection, thinking through the evidence and its possible implica- 
tions. Successful research in education also requires collaboration among research- 
ers and educators. Weak designs and measures, combined with frequent profes- 
sional doubts and disputes, have produced too many research results whose values 
and political implications are more prominent than their scientific validity.^ The 
Board’s conclusion in the earlier paper was, and remains, that more rigor is needed 
in education research.^ 

In fact, disagreements about what constitutes “rigorous” or “sound” research 
designs are continual and perhaps inevitable among prominent researchers. For 
example: 

• A 1999 conference at the Brookings Institution titled “Can We Make Education 
Policy on the Basis of Evidence?” examined the use of experimentation in 
education.® The panelists argued that only an experimental design in which 
individual students are assigned randomly— a design long used in medicine- 
will yield sound answers to questions from educators, policymakers, and par- 
ents about how to improve the practice and results of education. Randomized 
assignment designs were referred to as the “gold standard” for developing 
believable results that will be accepted for action by policymakers because, the 
panelists argued, there is little controversy over the findings when such designs 
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are employed. Research in education was described by one panelist as domi- 
nated by faculty in schools of education and motivated by craft principles, not 
scientific principles. Panelists lamented a failure of Congress to insist on such 
gold standard evaluations of the education programs they are funding and 
claimed a lack of leadership in the U.S. Department of Education to insist on 
them. 

• A contrasting perspective was expressed by Richard Murnane and Richard 

Nelson in a 1984 article in the Journal of Economic Behavior and Organization titled 
“Production and Innovation When Techniques are Tacit: The Case of Educa- 
tion.” * Their argument was that effective teaching requires experimentation 
and problem solving activity every day. Moreover, they asserted, while school 
authorities may provide the context in which teachers teach, they cannot 
control in any detail what a teacher does, either through monitoring or incen- 
tives. It is a mistake, the authors argue, to think of education research find 
development (R&D) in the same way as industrial or biomedical R&D. Educa- 
tion R&D should not be perceived as an expert-based activity that happens 
outside schools, or an effort to create “programs that work.” Instead, it should 
be part of the “problem solving, experimenting, evaluating, adapting to new 
contexts and goals that is always going on in education.” The appropriate way 
to ask about the influence of research on practice, they claim, is “what are the 
ways in which the new math and the modern physics ideas have influenced 
what goes on in classrooms, and in what ways, and in what contexts have these 
individual innovations enabled teachers to teach and students to learn more 
effectively.” 

Such differences among scholars about concepts of what research in education is or 
should he cannot be resolved by legislative fiats, but only through the questioning, 
responding, and revising cycle of the field as it addresses actual cases. There is a 
need to examine appropriate designs for conduct of research in education by 
drawing from many academic disciplines and research methodologies. That exami- 
nation should include insights that other fields of inquiry— such as biology and 
mathematics— can bring to illuminate the debate in education. Indeed, it should 
also involve policymakers and educators in order to deepen their understanding of 
how research may provide answers to their questions about education practice. The 
examination might produce a better understanding of what designs are appropriate 
for particular investigations than we now have, when vocal proponents of one 
approach or another overlook the larger context of learning and teaching, schools 
and communities. There is also need for rigorous explorations to find ways that 
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experimental methodology can be reconciled with the Murnane and Nelson per- 
spective. One means would be through randomization at the classroom or school 
level. Another would be to conceptualize and test other experimental or quasi- 
experimental methodologies that can strengthen findings about significant issues of 
lasting importance. 

In fact, recent OERI leadership has devised practical ways to advance understand- 
ing and use of appropriate research methodologies. The Assistant Secretary for 
OERI has initiated work through the RAND Corporation to create agendas for 
research in mathematics and reading that seek to build “balance” into future 
research plans. He has established Study groups of nationally recognized experts in 
these fields who will sift through what has already been learned, and formulate 
plans for additional investigations. The goal is to have a research program in math- 
ematics education and in reading education with rigorous research designs that can 
provide compelling, cumulative, and scientifically supportable findings. The designs 
for such research are to be balanced across all the studies in a program and appropri- 
ate for each separate investigation in a program. It is likely that the question of a 
balance in the mix of research methodologies will differ by field— that is, a “bal- 
ance” for research in mathematics might require more theory building, while bal- 
ance in reading might lend itself to theory-based “experimental” procedures that 
test the effects of specific models of instruction under particular circumstances. 

The Assistant Secretary is placing emphasis in other areas of OERI to advance the 
accumulation of research findings that document what has been learned (and so 
build credibility with educators), as well as develop supportable research plans. He 
has done this through collaboration with the regional educational laboratories and 
research and development centers, and in the Interagency Education Research 
Initiative (DERI), a partnership with the National Science Foundation and the 
National Institute of Child Health and Human Development (NICHD). In the 
latter area, lERI supports projects exhibiting “rigorous, interdisciplinary research on 
large-scale implementations of promising educational practices and technologies in 
complex and varied learning environments” to improve pre K— 12 student learning 
and achievement in reading, mathematics, and science. Projects are funded only if 
they meet tests for high standards of methodological rigor, sufficient scale, inte- 
grated use of technology, and conduct by interdisciplinary research teams. The 
program announcement states the research in these subjects “as a whole has lacked 
a convergent knowledge base that can inform systemic reform in a consistent and 
meaningful way. . . . Additionally, applicants are invited to take an existing body of 
research knowledge to the next level of investigation through efforts to extend such 
findings to complex educational settings.” 
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With these points in mind, Board members summarize their views as follows: The 
power of science comes from a combination of strong theory and data that bear on 
the theory. This implies endorsement of explicit ideas and agreed-upon methods for 
exploring and testing these ideas based on observation that has internal and exter- 
nal consistency. Experiments, as a classification of research, should not be 
scattershot or universal. Rather, they should be justified by a cumulative record of 
rigorous observation and piloting. This requires knowledge of context in addition 
to adherence to scientific canons. While experiments in education may not be used 
as frequently as they should as a preferred means for investigation— for a variety of 
reasons, but availability of funds is surely one such reason— “science” should not 
be equated with “experiments.” Nor should it be forgotten that a critical element 
of this cumulative record must be basic research that extends knowledge in topics 
central to teaching and learning. This approach may not solve problems immedi- 
ately, but would develop underlying “leading indicators” and intervening variables 
necessary to make progress toward solving problems. 

Members of the Board believe that actions of the OERI leadership as described 
above, and not a legislative prescription, will result in designs for research that have 
credibility and quality. The idea should be to create, in a program area, a portfolio 
of research that is designed to yield strong, research-driven findings with clearly 
stated implications for school and educational practice. Recommendation 1, then, 
calls for rigorous research desi^, appropriate for the matters being investigated. It would 
require a chief of research in the Department to make annual statements on 
progress toward this goal. 



Recommendation 2: Using research knowledge— 'Legislation should 
assign a major role to the chief of research in the U.S. Department of 
Education to conceptualize and carry out a program that connects 
research with practice so that student learning is advanced. The 
policy board (described in recommendation 6) should be charged 
with evaluation and monitoring progress under the plan for conduct 
of this strategy. 

This topic addresses how to foster use of research-based knowledge and applica- 
tions in school settings, something that has been a challenge as long as the federal 
government has invested in research in education. Traditionally, the “solution” has 
been to view the connection of research to practice in linear fashion: research 
produces knowledge, developers use that knowledge to create materials or “pro- 
grams” or training, and teachers/principals/superintendents and boards “imple- 
ment” the research findings and development products. 
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Over more than 35 years, the federal government has established institutions to 
close the gap between what we know and what we do in education. For example, 
the Educational Resources Information Center ^RIC) system, the Regional Edu- 
cational Laboratories, the Comprehensive Regional Assistance Centers, Regional 
Technology in Education Consortia, and Eisenhower Professional Development 
Federal Activities Program were created, at least in part, for this purpose. Despite all 
of them, there is a persistent finding that good research does not influence practice as it should. 

In gathering data on this issue, the Board commissioned several papers and re- 
viewed results from a conference on “Knowledge Mobilization and Use in Educa- 
tion” sponsored by the Assistant Secretary. Drawing on these resources and the 
broad experience of Board members themselves, the Board has identified three 
important elements for inclusion in a comprehensive dissemination strategy: 

• First, researchers must accept a responsibility forconducting their work in ways that will 
provide credible results, presented in understandable ways. They must reach out to 
educators. For example, one particularly important activity is development of 
syntheses of research around important and enduring problems facing educa- 
tors. Syntheses serve as a basis for technical assistance and to identify addi- 
tional needs for research. Teachers and administrators must participate directly 
in such syntheses, not only as users of results but as co-shapers of research 
questions, designs, and interpretations. The efforts must not just summarize; 
they must also reach judgments about the respective worth of various research 
reports using a best evidence approach. Researcher and educator collaboration 
will assure that the summaries can be relevant, timely, and well focused. 

• Second, schools— and the leaders and teachers in them— must be engaged in continuous 
improvement ^orts. This implies that educators will create a demand for the finding 
from research, as well as for access to the skills of researchers. Studies of effective 
improvements in school and classroom practice have demonstrated that re- 
search will be used only when demand for it becomes more sophisticated, and 
when teachers and other educational staff are players in its adaptation on site. 
Dissemination is better seen as a process, not an event. Continuing school 
improvement involves searching for what can be learned from others— often 
through “networks” of contacts among teachers and researchers; questioning 
how teachers’ work might lead to higher levels of student achievement; and 
data gathering, plarming, piloting, and revising what teachers and school leaders 
do. This is a distillation process, drawing on both practice and research. The 
Department’s research and dissemination efforts should encourage educators to 
use their own knowledge and that of others, and to reflect on what they have 
learned. 
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• Third, preparation for the profession of teaching needs to equip teachers to understand and 
use research. Preservice education of teachers has given virtually no attention to 
the understanding, use, and adaptation of research to enhance student learning. 
Yet such an understanding, and opportunities to practice continuing school 
improvement are essential to a new conception of “dissemination” not just as 
access to information, but as adaptation and use of findings from research. 

The thread that binds together these three foundations for a new conception of 
dissemination is '"collaboration “among researchers, teachers, and school leaders. Collabora- 
tion is the crucial common element from many current experiences that have come 
to the Board’s attention— for example, the Consortium on Chicago School Re- 
search, several school reform “networks” such as the American Federation of 
Teachers (AFT) research and dissemination network, a study of work of the 
regional laboratories, and a recommendation to the Board from the National 
Academy of Education for “design-based, problem solving research.” What many 
of these collaborations encourage is a willingness for all partners to learn from each 
other and to share responsibility for what is done. They often create a legacy of 
capacity building in the schools as well as a sharing of expertise and experience 
during the life of the collaboration. 

What is missing at the OERI level is an overall strategy to fashion these three basic 
elements into a coherent whole. Creating such a strategy cannot be done by legisla- 
tive mandate. A workable strategy for movingfrom dissemination to use of research needs 
to be an executive branch responsibility, because it must be developed with con- 
tinuing participation of many individuals and organizations, and because it will 
require a sustained effort over time, with many mid-course modifications. 

What is needed in the principal research office of the U.S. Department of Educa- 
tion is a capacity to conceptualize a program for moving research into practice. This 
wouldbeaprogram ofreseanh arid developmental activities that inform doe improvement of 
educatidnpracticearidpolicyfor greater sttident achievement Peer review panels, well- 
targeted development efforts, and relationships with consumer groups are required. 
The work should build cumulatively through collaboration that makes research accessible 
for educators and, at the same time, creates a demand for research. Recommendation 
2, in summary, calls for conceptualization of a program that connects research with 
practice so that student learning is advanced. The plans and activities to foster use 
of research, under that program, should be evaluated and monitored on a continu- 
ing basis. The Board proposes language in recommendations 2 and 6 for conduct of 
this oversight function. 
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Recommendation 3: Federal focal point for research in education— 
The U.S. Department of Education should have responsibility for 
support and conduct of research and development in education and 
for collaboration with other agencies. 

Members of the National Educational Research Policy and Priorities Board believe 
that the historic and basic purpose of the U.S. Department of Education is to 
support research and information collection, so that public discussion of education 
issues in America can be informed with facts and reliable analyses of data. 

Some individuals have advised Congress to sever research functions from the 
Department and place them in an independent agency. Proponents of this idea 
argue that education research and data collection cannot be protected from political 
influences, retain credibility for the public, or build stability so long as it remains in 
the U.S. Department of Education. However, they have not said how a small and 
single-purpose education research and data collection agency would be shielded 
from inappropriate external policy influences. Such an s^ency would still be a part 
of the federal government and subject to budget, data burden, and policy review by 
the Office of Management and Budget (OMB) and to Congressional legislative and 
appropriation processes. Board members are not persuaded that the case for excis- 
ing research functions from the Department is desirable, practical, or likely to result 
in strengthening the capacity for research. 

The U.S. Department of Education is the primary agency charged with federal 
leadership and administrative functions in education. It bears a responsibility for 
functions that have been vested in the federal government for 133 years. When the 
first federal authorization for a “Department of Education” was enacted and 
approved in 1867, its entire role was: 

“collecting such statistics and facts as shall show the condition and progress 
of education . . . and diffusing such information respecting the organization 
and management of schools and school systems, and methods of teaching, 
as shall aid the people of the United States in the establishment and main- 
tenance of efficient school systems, and otherwise promote the cause of 
education throughout the country.” 

In brief, it is not realistic that the U.S. Department of Education should give up its 
historic and fundamental responsibilities in research and data collection at a time 
when American citizens are demanding education services that will effectively and 
dependably lead to increased student achievement. 
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The view of Board members is that these responsibilities should be maintained in 
the U.S. Department of Education. Congress and the executive branch should 
concentrate not on relocating education research functions, but on strengthening 
and improving them by building the capacity of the Department’s principal research 
arm. This will require building up the staffing and funding of OERI so it can 
perform its necessary functions— supporting research efforts; collaborating with 
researchers, educators, and policymakers; linking research with practice; and 
cooperating with other agencies that have responsibilities for certain aspects of 
research in education. The Board’s other recommendations in this policy paper and 
its June 1999 predecessor are aimed at just that purpose. 



Recommendation 4: Effective structure and staff capacity— EWtm- 
n'ate organization structure and funding mandates in the Educational 
Research, Development, Dissemination, and Improvement Act of 
1994, and provide appropriate staff resources, so that units can sup- 
port critical masses of quality work, consistent with Board priorities: 

• Authorize the Secretary to form management groups around the 
principal problem-centered programs of the Department’s education 
research organization. 

• Alternatively, if the institute structure is retained from the 1994 
Act, the Board would establish a clear priority for institutes that focus 
on functions assigned to the National Institute on the Education of 
At-Risk Students, and the National Institute on Student Achievement, 
Curriculum, and Assessment. 

• The staffing resources of OERI must be strengthened so that 
responsibilities for planning, supporting, evaluating, and summariz- 
ing research can be distributed and adequately supported. 

• The current funding allocation requirements should be removed. 
Provisions that earmark 25 percent of institute funds for field-initiated 
studies and one-third of all institute funds for university-based re- 
search and development centers have constricted sound plans for 
focusing on a limited number of research priorities. There are other 
funding prescriptions that should also be removed, including ones for 
institutes, coordination and synthesis, regional laboratories and rural 
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areas within them, research and development centers, the Educa- 
tional Resources Information Center (ERIC) Clearinghouses, and 
field-initiated studies. 

In its June 1999 policy recommendations, the Board proposes that the OERI 
mission and resources be aligned and that appropriation allocation set-asides and 
mandates for institutes and types of support be removed from the 1994 Educa- 
tional Research, Development, Dissemination, and Improvement Act. The Board 
found an extreme mismatch between the available resources for the institutes 
compared with what was contemplated when problem-focused institutes were first 
conceived. It suggested that “it would be prudent if the Secretary had authority to 
modify the organizational structure” to bring about a better alignment. It also 
pointed out that the funding allocation rules frustrate responsiveness to new needs 
and circumstances, fractionate already limited funds, and inhibit response to new 
national priorities.^ 

The circumstances that led to these Board recommendations have not essentially 
changed in the last year. There is not a sufficient level of funding that can permit 
five institutes to support adequate levels of research or staff to be involved in 
significant ways with the field. If anything, the situation is more dire today. Board 
members are not opposed, in principle, to the categories for institutes as Congress 
created them in 1994, but believe that the realities of the situation must be faced. 

Board members believe that the most compelling approach is to eliminate specific 
internal structures from the authorization la^S^. Instead, there should be authority for 
the Secretary to create problem-centered management groups that would bear 
responsibility for conduct of priority activities for so long as they are needed. This 
would accommodate the current priorities for reading and mathematics initiatives, 
allow them to be refocused or eliminated over time, as appropriate, and permit 
creation of new management groups as priorities and resources make possible. 

However, if the current legislative specification for institutes is retained, the Board 
would prefer stronger alignment of program priorities, funding, and missions 
assigned to the institutes. For that reason, the Board would identify the focus of 
missions for two of the 1994 Act institutes as organizational homes for the priori- 
ties described in its June 1999 recommendations: a high level of achievement for all 
students, initially emphasizing readingand mathematics.^ Those would be the National 
Institute on Education of At-Risk Studerits and the National Institute on Student 
V Achievement, Curriculum, and Assessment. 
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The Board speaks specifically about the critical need for rebuilding OERI staff in 
its previous policy paper. It describes the need for staff who can participate as 
peers in the scholarly community and work with users to facilitate the practical 
application of knowledge. It notes the leadership roles that staff play in other 
federal research agencies to advance the enterprise and, at the same time, attract 
and retain highly trained and capable individuals— roles such as conduct of re- 
search, synthesis of research, collaboration with external stakeholders, planning 
and designing cutting-edge research, organization of and participation in peer 
reviews, review of proposals for social utility or agency relevance, and evaluation 
of ongoing research. 

OERI staff capacity and responsibilities have gone in opposite directions. Since the 
OERI reauthorization was enacted in 1994, OERI staff have declined from 373 to 
324, and the leadership Senior Executive Service positions have dropped from 9 to 
5. Over this same period, a single Office of Research has been divided into five 
separate institutes, and new responsibilities have been added, such as the expansion 
of technology programs and interagency initiatives. There is an important role for 
professional staff in a research agency. There is no substitute for it, nor any excuse 
for continued failure to address the problem. 

The issue addressed in recommendation 4 is practical administration and account- 
ability. The existing law creates expectations and structures that simply cannot be 
fulfilled under the current circumstances. The effects of lack of funds and staff are 
immediate and real. Statutory and management changes that will bring expectations 
and resources into better alignment— and do so around the Board priorities for 
learning by all students, especially in reading and mathematics— are urgently 
needed. 



Recommendation 5: Stability and professionalism in directing re- 
search— h. chief research officer for the U.S. Department of Educa- 
tion should be appointed by the President for a 6-year term and 
should report directly to the Secretary. 

In its 1999 recommendations, the Board proposes that the Congress create a focal 
point for research leadership that can span across administrations.’ The paper 
argues that the issues of quality across the agency, coordination of work internally 
and collaboration externally, and the substantive development of the research 
agenda call for continuing supervision, mentoring, and quality review. It would be 
highly desirable to build into the Department the means for stabilizing a profes- 
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sional research function that offers some insulation from constant changes in 
leadership and course of direction, even appearances of politically inspired or 
ideological research agendasd° 

Board members have considered the testimony of several individuals before con- 
gressional committees, or of others who made their recommendations in writing, 
that concur with the need for such a position. Some suggest a “commissioner” (in a 
form similar to that of the National Center for Education Statistics), appointed by 
the President, but located in OERI alongside the National Center for Education 
Statistics (NCES). Witnesses have called for appointment of individuals with 
distinguished backgrounds in research and development or in evaluation. Several 
have suggested presidential appointment. 

The Board recommends creation of a chief research officer position in the Depart- 
ment, one that would require appointment by the President and confirmation by 
the Senate, and provide a 6-year term of office. The incumbent would report 
directly to the Secretary. There would no longer be an Assistant Secretary appointee 
serving at the pleasure of the President. This arrangement draws from the 
administration’s proposal and also from precedents in other scientific agencies, such 
as the National Science Foundation, but adapts them to the responsibilities of the 
U.S. Department of Education. It would provide a critical measure of leadership, 
professionalism, independence, continuity, and stability long needed for the 
Department’s research activities. It would provide a visible place for reseiarch in the 
Department and direct access to the Department’s decisionmaking and policy 
setting functions. 

Others might argue for retention of an Assistant Secretary as well as a research 
chief. For example, the U.S. Department of Commerce has an Under Secretary for 
Economic Affairs appointed to serve at the pleasure of the President. Within that 
office are a presidentially appointed Director of the Census, who also serves at the 
pleasure of the President, and a senior level career chief for the Bureau of Eco- 
nomic Analysis. Board members are not convinced that such an arrangement would 
be appropriate for leadership of research functions in Education. Members believe 
that adding a presidentially appointed research chief, while retaining the assistant 
secretary position, would result in confusion of responsibilities and could lead to 
work at cross purposes. It would, by its nature, keep the research chief— and the 
research function— in a subordinate position as compared with the principal offic- 
ers of the Department, rather than reporting directly to the Secretary, as a primairy 
Departmental officer should. 
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Recommendation 6: Policy board appointments and functions— 
There should be a policy board comprised of policymakers, educa- 
tors, researchers, and the general public. Members would be ap- 
pointed to 6-year terms, initially staggered. Board responsibilities 
should include setting policies, reviewing conduct of the agency, 
fostering and overseeing collaboration among federal agencies that 
conduct research in education, advising the chief research officer of 
the Department and the Secretary, and reporting to the public on the 
condition of the agency as well as that of education research in the 
nation. 

This area of recommendations is perhaps most difficult for Board members to 
address, simply because of their status as current appointees. Nevertheless, the 
Board is making recommendations because some members of the public have 
asked, and because its members have now had considerable experience with the 
1994 legislation, so their reflections on that experience should be useful to others. 

In statements before Congress, and in other public proposals for OERI reauthoriza- 
tion, there have been a variety of suggestions about the nature and responsibility 
of public bodies attached to the research operations functions. Some have pro- 
posed a board with membership similar to that set out in the 1994 law, including 
researchers, school-based professional educators, and public members “knowledge- 
able about educational needs of the United States.” Others have proposed a board 
modeled after the National Assessment Governing Board (NAGB)— which sets 
policy for the National Assessment of Educational Progress— with representation 
of different political parties and specifically including governors, legislators, chief 
state school officers, representatives of business and industry, and the general 
public, including parents. Appointment by the President has been proposed, as well 
as appointment by the Secretary, which would continue the 1994 provision. There 
have been several suggestions either to retain or alter the Board’s responsibilities 
under that law. 

The Board is not strongly concerned about whether appointments are made by the 
President or the Secretary. Both approaches have precedents in the Department, 
and either can work. An argument favoring presidential appointment, as the admin- 
istration has proposed, is that approach is frequently followed in other research 
agencies. Secretarial appointment, as under the 1994 law, can sometimes better and 
more quickly respond to broad agency interests. 



Investing in Research 




61 



A Second Policy Statement with Further Recommendations for Research in Education 



Members have decided opinions, however, about the composition of the Board. 
The Board should be above narrow partisanship, represent a diversity of perspec- 
tives, and reflect the good judgment of whoever has responsibility for appoint- 
ments. Clearly it is possible to create a strong and representative board with ap- 
proximately equal representation of researchers'', practicing educators from all 
levels'^ and other members, of the public", as under the current appointment 
categories. But there could be a much enhanced visibility and continuity for educa- 
tion research if individuals representing public officials, specifically including 
different political parties, were regularly a part of the Board’s conversations. For 
example, including among the members two governors, each from a different 
political party; two state legislators, each from a different political party; and two 
other elected policy officials would provide this kind of visibility and enrichment 
of Board discussions. While the current Board’s 15 members have usually been 
sufficient, the Board would find it appropriate to authorize an increase in the 
membership of a future education research board if there could be greater partici- 
pation of public officials in it. 

Finally, as to the functions of such a group. Board members would classify these in 
policy setting and prioritizing terms, as set out in the 1994 Act. The Board would 
endorse separate responsibilities for the new board, on the one hand, and the chief 
research officer on the other. The new board would not have any direct authority 
over day-to-day management, appointments, execution of policies, or setting the 
leadership tone for conduct of education research. It would, however, play the 
following roles: 

• establish policies on priorities for education research investments and building 
capacity in education research, and on standards under which that work is con- 
ducted; 

• review conduct oi the agency, including documenting, analyzing, and judging 
agency actions under board policies and also reviewing awards of funds, either 
in the formulation stage or after completion of the work; 

• monitor and evaluate the conceptualization and conduct of the chief research 
officer’s program to connect research with practice so that student learning is 
advanced; 

• foster and oversee collaboration among federal agencies that conduct research in 
education, and establish broad policies for the U.S. Department of Education 
role in relation to that of other agencies conducting education research; 
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• advise the chief research officer of the Department and the Secretary on education 
research; and 

• report to thepuhlic on the condition of the agency and the progress of education 
research in the nation, drawing from the report of the chief of education 
research on the state of education research. 

These are appropriate areas of responsibility for a Board whose purpose is to 
participate in setting directions and also to bring credibility to education research. 

A board performing these roles, with the long, overlapping, and diverse appoint- 
ments recommended here, can also serve to make education research more stable 
and continuing. Whether the Secretary or the President will succeed in attracting 
exceptional individuals to a future research board will depend on the nature of the 
charge to that Board. The responsibilities must be of sufficient magnitude and 
complexity to make service on the board worthy of the. prospective members' time. 



1. The Board’s first policy recommendations paper, Investing in Lmming:A Policy Statement withRecommen' 
dations on Research in Education^ was published in June 1999 . Copies are available from the office of the 
National Educational Research Policy and Priorities Board, Telephone: (202) 219^2324, fax (202) 219- 
1528, email: waiiser_green@ed.gov, or consult the NERPPB Web Site 

(http:/ / www.ed.gov/ offices/ OERI/NERPPB/). 

2. Ibid., p. 27. 

3. Ibid., p. 8. 

4. Ibid., p. 17. 

5. Transcript of a Brookings Press Forum, “Can We Make Education Policy on the Basis of Evidence? 
What Constitutes High Quality Education Research and How Can it Be Incorporated Into 
Policymaking^” Wednesday, December 8, 1999; Tom Loveless, The Brooking? Institution, host; Paul 
Peterson, Harvard University, chair. The sponsors of this conference sought to advance use of random- 
ized assignment of participants in “controlled experiments” as the preferred design for education 
research. Randomized experiments were described by several panelists as the gold standard for research. 
The transcript indicated the following: 

Only serious experimentation will yield the answer^ needed to improve American education. 
Use of randomized assignment research design has most frequently been publicized in 
medicine, but has been the basis for important studies in social policy areas such as employ- 
ment and welfare. Sometimes education research is designed this way, but not very often. One 
panelist cited a study of 1200 research reports in the AERA journal over a 30-year period 
turned up 31 experiments from the 1200 studies. While no attempt was made to reach 
conclusions from the conference, individual panelists expressed their opinions, one of which 
that “people here in Washington who think about how to write our laws and administer our 
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laws” should ponder this. Public policy can be influenced by randomized experiments almost 
more certainly than any other research strategy. The randomized experiment has a capacity to 
influence public policy because there is much less controversy about what the finding is. It was 
asserted that randomized experiments increase the probability of use of research results 
compared with other forms of research. 

Some instances in which randomized experiments have been employed to yield compelling 
findings in education include micro experiments to field test how children watch television, and 
how distraaible they are for development of Sesame Street; the well-known Perry Preschool 
Projea with Head Start children; the Philadelphia White Wing Foundation randomized 
experiments on the effea of awarding small scholarships to children from poor families who 
achieve at high levels in school; and introduction of standardized testing in Irish schools. 
Several speakers argued against some of the reasons frequently cited to explain why random- 
ized experiments are infrequently used in education — especially emphasizing a culture of 
research in schools of education that has evolved over the past three or four decades, which 
were described as knowledge growth based on craft principles and not scientific principles. 

These include: a belief that experiments can’t be done in schools; a belief that experiments are 
unethical because they involve withholding potentially a share of educational practices from 
some children who might need them; a belief that educational interventions can’t be standard- 
ized; there are better methods for evaluating educational interventions; governments and 
foundations have not really pressed for quality evaluation of what works in education. 

One panelist set out several conditions when randomized assignment experiments are the 
appropriate research design, including: if one wants to know whether the program or reform 
makes a difference; whether the program under study is sufficiently different from business as 
usual; when participants in the research are not being denied access to an entitlement; when an 
important question is being addressed; when you can get cooperation; and when you have the 
resources and ability to do a quality study. To do a good job on randomized experiments, for 
example in the Title I compensatory education arena or Head Start, takes $5 to $ 10 million 
experiments. 

6. Richard J. Murnane, Harvard University, and Richard R. Nelson, Yale University, in the Journal of 
Ecorwmk Behavior and Organization 5 (1984J *Troduction and Innovation When Techniques are Tadt: The 
Case of Education.” In an early 1980s study of input-output research as applied to education, Richard 
Mumane and Richard Nelson observe that variation in education practice is unavoidable and in fact is 
crucial to effeaive teaching. Their analysis bears direcdy on the conditions for conducting research in 
education: 

Effective teaching requires intensive problem solving activity, and creative and personalized 
responses to frequent unpredlaed circumstances. It is dear that this interpretation, which we 
bdieve is the correa one, casts a shadow on the faith that what one teacher or school is doing 

with success, another can replicate with comparable effea school authorities, while able to 

provide the context within which teachers gp about their jobs, they cannot control in any detail 
what a teacher does either through monitoring or through incentives. . . . Teaching, if it is to be 
done effeaively, involves e3q)erimentation. Some children learn rapidly, others slowly; what is 
effeaive for one may not be effeaive for another. From time immemorial teachers have had to 
find out for themsdves what works with which children and with which subjea matter; 
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... it is a mistake to think of educational R6cD as like industrial or biomedical R8cD. We think 
it is also a mistake to think of Innovation in education exclusively, or even largely, as an activity 
conduaed in specialized facilities by specialists in R6cD. We also believe that it is inappropriate 
to judge the contributions that particular innovations have made to educational practice by 

surveying the extent to which particular sets of new blueprints are in use in different sites 

In summary, we believe that educational R8cD should not be viewed as creating "programs that 
work,” but rather as part of the problem solving, experimenting, evaluating, adapting to new 
contexts and goals, that always is going on in education. . . . Educational R6cD provides a flow 
of ideas, broadly defined methods, evidence about what is being tried out in different settings 
and about how well particular initiatives have worked in these settings, that enrich capabilities 
for the experimentation and problem solving that go on in individual school systems, schools 
and classrooms. 

Thus, the relevant question is not "how widespread is the use of the modern physics package, 
or the new math package, and what have been the effects of use of these packages on perfor- 
mance.” Rather, one might ask “what are the ways in which the new math and the modern 
physics ideas have influenced what goes on in classrooms, and in what ways, and in what 
contexts have these individual innovations enabled teachers to teach and students to learn 
more effectively.” . . . What will work and what will not work varies from situation to situation. 
Much problem solving and fine tuning inevitably must go on in the particular school and 
classroom, and thus what someone else has done successfully can provide only gross guidance 
as to what might (or might not) be effective in a different context. 

7. Investing in Learning, p. 41. 

8. Ibid., p. 21. 

9. Ibid., p. 42. 

10. Ibid., p. 41. 

11. The Board includes in the definition of researchers those from the social and behavioral sciences and 
other fields who can make important contributions to student learning. 

12. Educators include teachers and administrators at all levels of education. 

13. Other members in the current Board appointment categories are: "parents with experience in 
promoting parental involvement in education, chief state school officers, local educational agency 
superintendents, principals, members of state or local boards of education or Bureau-funded (that Is, 
Bureau of Indian Affairs) school boards, and individuals from business and industry with experience in 
promoting private seaor involvement in education.” 
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